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describing, analyzing, and summarizing evaluation data. Chapter 4, which 
begins with the proposition that workplace education professionals are 
accountable for all aspects of workplace education program planning from 
inputs to outcomes, provides guidelines for reporting evaluation results, 
documenting results, analyzing how programs have affected companies, and 
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Preface 

Evaluation is, first and foremost, a statement of accountability to the stakeholders of the program. 
Given that the primary stakeholders are the educational institution, the learners, and the 
companies served, each of these entities deserves and needs to know the kinds and degrees of 
change that have occurred as a result of the educational intervention that has been implemented. 

The educational institution must know whether the results of their input of people, resources, and 
managerial practices translates into a behavioral product that reflects well on the institution's 
ability to deliver high-quality learning activities. The learners must recognize that the knowledge 
gained as a result of the learning activities is demonstrable and of value to them in at least one 
area of their lives. The companies must be satisfied that the behavioral changes that have come 
about after the learning activities have taken place are beneficial to it in some defined way. 

Evaluation has proven to be a complex factor in workplace education programs. Some evaluators 
only want an indication of whether or not the learners themselves were satisfied with the 
instruction that they received: was the instructor effective? was the classroom comfortable? 
were the learning demands at a level where they were neither too simple nor too challenging? The 
answers to these and related questions are elicited from the learners' affective domain. This 
information can give evaluators a sense of the learners' comfort with the general tone of the 
learning activities. 

Some evaluators attempt to measure actual gains in learning. In a purely academic setting, the 
learners are, in fact, a captive audience. In these circumstances, the instructor may, at his/her 
discretion, give pre-tests. Differences in scores between them and subsequent post-tests may be 
able to give the instructor a measurable index of any increase in specific skills, given the 
assumption that the curriculum and the testing are correlated. A workplace situation has other 
problems. The workforce is usually less than enthusiastic about written pre-tests because they 
may be unwilling to demonstrate the level of their basic skills on the chance that the results may 
influence their job longevity. If the pre-test involves specific hands-on workplace skills, they may 
perceive the testing as useless since, if their workplace skills were not adequate, they would lose 
their jobs. So often in workplace education situations, quantitative measures may be difficult to 
come by. 

Indicators of retum-on-investment (ROI) have been notoriously difficult to establish. Direct links 
from the instructional setting to behavioral differences on the job have been tenuous at best. At 
the same time, these are the indicators in which the companies display the most interest. Last but 
not least, some evaluators want to have a sense of the educational program's effect on the larger 
community, how many workers have kept their jobs? how many have moved from a dislocated 
status to an employed one? what effect has the difference in job income had on the family unit? 

Another grand difference between an academic setting and a workplace education one is the 
nature of the curriculum. In a traditional educational setting, the curriculum is built upon the 
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content. The instructor, as the resident expert, determines what is to be learned and the depth of 
knowledge that is required. Normally there are time constraints on the learning of the material. 
The learners, in this instance, take a subordinate role. In a workplace setting, the content experts 
are the learners themselves. They, and company representatives, usually determine the 
curriculum. The instructor becomes the educational designer and arranges learning experiences 
that meet the goals of both the learners and the company. Each class and each group of learners, 
therefore, undergo a different experience. As each learning activity is customized, each 
evaluation needs to be customized, in order to faithfully represent the needs of all of the 
stakeholders. 

With the proliferation of workplace education programs, many adult education professionals have 
found themselves wearing many hats. Not only are they expected to develop programs, conduct 
needs assessments, write curricula, and train and supervise instructors; but they are asked to 
develop and implement an evaluation plan. In some cases, these hard-working and talented 
people have not had extensive training in evaluation procedures. It is my hope that this Manual 
may be of some use to them. 

In 1985, Dr. Edgar J. Boone, at the North Carolina State University, published a book on adult 
education programming ( Developing Programs in Adult Education ). His conceptual approach to 
the programming process has stimulated my thinking on this Evaluation Manual for workplace 
education practitioners. As his thinking has served as a source of inspiration for me, I am hopeful 
that this Manual may be of use not only to my institution but to other fledgling program 
evaluators in the workplace education field. 
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On the Road to Evaluation 



You've begun a long and slippery path. Keep remembering that you need all the help you can get. 
And there IS help out there. There are many fine books, articles, and publications on workplace 
education evaluation, and I encourage you to read as many of them as possible. However, for the 
moment, this manual will get you through those first steps. It is our hope that it will keep you 
headed in the right direction. 

OK. So first, what do you mean by the term "evaluation"? Write it down. 



What does your institution (let's assume that it is a community college) think that evaluation 
means? Your supervisor? The president? The governing board? Are the answers the same as 
yours or different? 



How about the industries and companies that you are working with? Or, in some cases, are 
hoping to work with? What does it mean to them? Again, are the answers similar to the others 
or are they different? 



Why are the answers to these questions important? 

Because the answers will determine the approach you use to evaluate the program. 



Overall questions to keep in mind during the entire evaluation process 

What is it that you want to measure? 

Who (of the above stakeholders) wants to measure what? 

What instruments are you going to use to measure? 

What will the interpretation of the resulting information mean? 
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Chapter One 

In our program, we begin with certain assumptions about what we are doing in workplace 
education. We think that these assumptions might hold true for most programs. However, they 
are flexible enough so that workplace education programs would be able to fit their program goals 
within them. 

Assumption Number One: 

The purpose of workplace education programs is to effect change in a group of workers. 

These changes are behavioral in nature: they can prescribe change in workers' knowledge base, in 
their ability to perform particular hands-on tasks, and/or their ability to work effectively with 
others in achieving job goals. 

The group of workers may be at any level: workplace professionals may be asked to develop 
training programs for management as well as rank-and-file workers. We may be asked to train 
members of a certain department or do a company-wide training project. In some cases, we may 
be asked to provide training for a number of plants, all of which are components of one company. 

Nevertheless, under any or all of those conditions, evaluators must clearly keep in mind that the 
overall goal is change. 

Assumption Number Two: 

The outcomes of the workplace education program must be clearly identified and 
evaluated. 

In other words, evaluators, you need to think about what exactly you are looking for as evidence 
of change (do the workers know the new formula, are the new teams accomplishing their goals 
more efficiently, can the workers machine parts faster and more accurately than before?). Then, 
to what extent has the change taken place (most of the workers know half the formula, new teams 
are using more resources but they like each other better, the scrap rate has remained the same at 
the machine shop). 

Assumption Number Three: 

Decisions about the workplace education program are rational and based on values. 

Each company has an organizational culture. That culture is based on a set of core values. These 
values may be ubiquitous throughout the company, or there may be sub-sets of values found in 
departments, levels, or professional groups. Workplace educators have a responsibility to help 
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the company make these values explicit. Programmatic choices will be made on the basis of these 
explicit values or sets of values. If the company's primary value is profitability, program choices 
are based on that assumption. If the company's primary value is employee morale, program 
choices (and evaluation) will show that concern. 

Assumption Number Four: 

Management of the workplace education program depends on program outcomes. 

Program outcomes generate a feedback loop to the original goals of the program. Each time an 
outcome is evaluated, decisions as to whether to continue, to suspend, or to modify the program 
are made by workplace educators and company personnel. So it is obvious that not only the 
success but the continuation of the program is dependent on the evaluator. 

Assumption Number Five: 

Workplace participants' response to the training is needed and valuable. 

The experience of the training and the people to whom this training is presented are not a duality. 
Together they constitute the event that we call the learning experience. The impact of that 
experience may or may not be a surprise to the evaluator. We need to know if the training was 
given at the appropriate level for comprehension, whether it was pertinent to the program's goals, 
or whether there were factors in the experience itself that generated outcomes that the workplace 
educators did not expect. 

Assumption Number Six: 

Workplace educators have a responsibility to their institution, to companies, to workers, 
and all other constituencies to account for the results of their programs. 

This may be too obvious since all of us are conscientious professionals. However, there may be 
times when outside pressures or just the distractions of everyday life may cause us to find 
ourselves coordinating a series of activities which have little or no relation to a coherent program. 
We are going to avoid that danger. 




2 



Chapter Two 

In this chapter, we are going to talk about what some professionals think that evaluation is, what 
its relationship is to other parts of the program, and what it is intended to do. 

Go back to the very first task we did — that is, definitions of evaluation. Review them. To what 
extent does your definition and the definitions of your colleagues agree or disagree with the 
following: 

♦ determining the extent to which objectives have been attained 

♦ the above criterion PLUS the amount of movement TOWARD the objectives 

♦ getting useful information about the program so that appropriate decisions can be made 

♦ determining how the curriculum and the instruction are accomplishing the objectives 

♦ finding out what the program "is" as opposed to what it "should be" 

♦ judging the value of a program as expressed by its social interactions and outcomes 

For example, Company A had given you a list of performance standards that workers in 
Department 101 should achieve. The training is over. Your evaluation can show how close the 
workers have come to these standards. 

Community College B is not sure whether or not to put more resources (personnel, space, 
computers, etc.) into its workplace education program; your evaluation gives the College the 
bases on which it may make the decision. 

You look at the effects of the workplace education program at Company C. Then you compare 
these effects with the program's original needs assessment. 

At Company D, you compare the workers' reaction to the program with the measured outcomes 
of training. 

Confusing? Yes, it is. But you also have the flexibility to decide what kind of evaluation is best 
for your institution. 

Warning: Whatever definition of evaluation you choose, consider that it should contain the 
following themes: 

It is an orderly and rational process. The business, culture, and society provide the framework in 
which you operate. There is no beginning or end to evaluation: it is ongoing as long as the 
program is in existence. The main product of evaluation is decision-making. Judgments must be 
based on criteria. 
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A Path Through the Wilderness 



You have a program to evaluate. Where do you begin? Here's one possible scenario that could 
work for you. Think of it as a map through some woodsy area that you are not terribly familiar 
with. 

Look at what the program is about. Somewhere there is a plan of action, perhaps several. 
Somewhere there is a list of objectives, probably several. The program activity is centered around 
a teaching-learning experience that is facilitated by an instructor. The objectives, hopefully, are 
behaviorally based. If you have access to these documents, you are home free. 

Step One. Evaluate the success of the program at the individual level. This is where the 
spotlight falls; has the behavior of the target group changed? 

Step Two. Evaluate the major objectives of the program. Are these objectives based on need 
(companies, workers, other stakeholders)? Were these needs expressed (i.e. worker group 
indicated that they did not know the math to perform a task) or were they analyzed (i.e. the 
company is planning to move to a teamwork system and wants its workforce knowledgeable 
about it)? 

Step Three. Evaluate the linkage between the needs instruments and what is going on in the 
teaching/learning activity. Is there an interconnect? 

Step Four. Evaluate the impact of the environment on the program (i.e. are there management 
policies that affect the program's success?) 

Step Five. Compile your information so that it can be of benefit to the program's decision- 
makers when they contemplate what to do next. 

If you use this schema, you will find that evaluation activities link you to all the parts of a total 
program. You may find that the workers are not learning because the material being presented is 
either too easy or too difficult for their level. You may find that it is almost impossible to come 
up with any useful data for the company because the objectives of the program were not clear or 
measurable. Expected resources may not have materialized or were of a poor quality. The 
program may have been well-designed and appropriately delivered, but marketing efforts were 
poor and did not reach the target groups. All of these concerns will reflect different parts of the 
program that may need to be fixed before the next cycle of activity is undertaken. 

Now you have the raw data for your next step. 





The intention of evaluation is to gather the information that is needed to answer the levels of 
questions that you already have become familiar with. 

When you are looking at individual behavioral change, you can measure objective learning by 
testing workers, by supervisor ratings of workers, or by workers' self-ratings. You can include in 
your investigation such questions as whether they learned what they wanted to learn, whether 
they learned things they didn't expect to, whether they can use what they learned in another 
environment (home, volunteer work), whether the learning was put to use immediately or whether 
they expect to use it somewhat later down the line in the workplace. And we can include a smile 
sheet also to make sure that unexpected bad things were not happening that we were not aware of 
or were too distracted to ask about. 

Looking at aggregate behavior change, you can use simple descriptive statistics if you have the 
kind of data that are quantifiable. If not, you can create a summary sheet delineating the general 
thrust of the data. In other words, you can use the same data base as above, but, instead of 
reporting the results on an individual basis, simply report it as group totals. 

It will be much trickier to try to specify the relationships between program activities and 
behavior change. For example, if one of the goals of the program is to teach the participant to 
change a tire (and the curriculum on Tire Changing has been implemented), you may need to 
follow up with some other questions. Has the participant had any other training or experience in 
tire changing before this training took place? Has he/she changed tires in any other setting? Have 
any friends or family members helped him/her in changing tires outside of the training site? In 
other words, you will have to think about what other factors could have had impact upon the 
results of the training. 

When you consider the relationship between the organization and program outcomes, it will be 
an opportunity for you to address how more general factors had an impact on the training. For 
example, was the organization supportive of the training? Did that support come from both the 
formal and the informal structures? What effect did the training have on the organization? Did it 
disrupt work patterns? Did the participants' social roles undergo a change? 

By this time, you should have a good handle on what you want to include in your final product. In 
fact, you may have a rough draft. The next chapter will get you through the nitty gritty. 
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Chapter Three 

In this chapter you will find a definition of evaluation that is practical, concise, and useable. Also, 
some notes about how objectives should work and hints on describing, analyzing, and 
summarizing your data. 

Evaluation is a coordinated process carried on by the total system and its individual 
subsystems. It consists of makin g Judgments about planned programs based on established 
criteria and known, observable evidence. 

Boone, E.J., 1985, p. 179 

What do you think are five things that you really need to do when evaluating a workplace 
education program? (We’ll go over them throughout this chapter.) 

1 . 

2 . 

3. 

4. 

5. 

•srFirst of all, you absolutely positively must describe the outcomes of the program! 
(Hopefully, this appeared as one of your statements above.) 

As we discussed in the previous chapter, the outcomes could have been immediate (yes, the 
participants can now fill out technical reports accurately) or later (the incidence of workplace 
violence has decreased by 30% in the year following Conflict Management training). The 
outcomes may have appeared in the program's objectives (reading levels went up one full grade) 
or they could have been a complete surprise (absenteeism has dropped 1 5% since the basic skills 
classes began). Which leads us directly to the issue of 

©Objectives ® 
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We remember that program objectives are derived from the needs assessment. The needs are 
translated into the objectives that you are in the process of evaluating. Now, objectives can be 
broad (macro) or specific (micro). For example, a small machining company is concerned about 
time loss in production because many of the machinists seem to have difficulty reading the 
blueprints and spend time off the floor getting that information from the supervisor. Not only is 
production time lost, but the supervisor is distracted from other duties to attend to this problem. 

The machinists are assessed through individual interviews and through short diagnostic instruments 
which measure their skills in spatial relations, mechanical reasoning, math, and reading. The 
supervisor is interviewed for information concerning the amount of time he/she spends in 
interpreting the workers' queries on blueprints. 

The macro objective is to cut down the time workers spend asking questions about blueprints. 

The micro objectives are to increase workers' skills in the above four areas. The assumption is 
that the upgrading of these skills will impact on the quality of blueprint reading. 

Macro and micro objectives can be treated like Chinese boxes, with one fitting into the other. 
Evaluation of the micro objectives can give you good information about the success of the macro 
objective. 

In addition, you may find it useful to evaluate broader objectives of the program, such as the 
company's overall objectives (raising the machine shop's productivity rate) or community 
objectives (piece work rates go up, thereby bringing more income to the area). 

Think of this whole process as a hierarchy, with the objectives at the individual level feeding into 
larger programmatic objectives, and all fitting into long-range company and community goals. 

Now that we have our conceptual map in place and we know where we want everything to fit, it's 
time to look at how to describe the various inputs and outcomes of the program. 

The three activities that compose the act of evaluative description are: 

...collecting 
...documenting and 

...communicating what the outcomes of the program are. 

First, let's talk about the outcomes. (You can also call them “outputs” or “results.” It doesn’t 
matter.) You can arrange for outcome indicators that the workplace participants are aware of or 
take a part in. These can be interviews, questionnaires, tests, etc. The workers know that they are 
involved in the process of telling you something about the educational activity, either quantitatively 
(testing) or qualitatively (smile sheets). 
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Then, there are indicators that they may not be aware of. You may wish to set up instances of 
direct observation either during training or in the workplace. How many times has machinist A 
asked his/her supervisor about an item on a blueprint this week? How many requests for 
information on blueprints has the supervisor answered this week? How do these rates compare 
from week to week over the training period? How many participants in the training program have 
asked about further training? 

Both intended and unintended outcomes are valuable. However, intended outcomes should be 
derived directly from the program’s objectives, and, hopefully, they exist in written form. These 
outcomes can then be assessed quantitatively. The compilation of these kinds of data are critical to 
the development of credibility and accountability with the program’s stakeholders. And we all 
know that if our program loses credibility and/or accountability, we are not performing the 
professional job that we are capable of. 

Unintended outcomes can best be presented in a qualitative form. A description or a case-study 
format would be more appropriate for presentation purposes. 

For example, going back to our machining company, you may find that: 

1 . 40% of the participants completed Objective 1 . 

2. 99% of the participants completed Objective 2. 

3. 23% of the participants completed Objective 3. 

In an attached description, you may report that the participants thought that Objective 2 was a 
waste of time because they already knew and used that skill on a daily basis in the workplace. 
(Hmmm, the needs assessment instruments may need to be reviewed!) The instructor reported that 
Objective 3 was difficult and frustrating for the participants because the level of their math skills 
was not high enough to accommodate the material. The class was quite interested in the skills to 
be learned in Objective 1 because it involved learning new technology that was soon to be coming 
into the plant; however, the instructor seemed unsure of how to teach it, and, by the way, the 
software for the program never did arrive. 

Both the quantitative and the qualitative data are valuable resources in helping program decision- 
makers design the next phase of the training. 

If you have enough time left in the program, it might be useful for you to arrange a follow-up 
study after six months or one year to find out to what extent the participants are using their new- 
found skills appropriately. That’s right, no rest for the evaluator. 

OK, we have talked about outcomes. Now, let’s talk about program inputs. 

Whose inputs are they, anyway? You need to consider all the resources that go into the 
program from all the stakeholders. The community college may be supplying the instructors, the 
curriculum, and the supplies; the company may be supplying the facility; the workers may be 
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investing their own time. Everything that goes into the program should be documented! Why? 
Because at some point, you are going to have to try to make the connection between these inputs 
and the outcomes of the program. So try to be as concise and as concrete as you can. 

What is each group of stakeholders contributing toward the goals of the program? 

At a minimum, a case could be made that there is a correlation between what the stakeholders put 
in and what they get out. At a more sophisticated level, look for evidence of covariation. For 
example, you may find that for each hour of instruction up to forty hours, workplace math skills 
increase one level per two-hour class segment. After forty hours, math skills plateau, and further 
increases drop to one math level per six hours. With this kind of information, the program 
decision-makers can design workplace math skills programs that can stay within certain cost and 
time limitations. More later. 

You are now ready to describe program inputs and outcomes. What forms can you use? As we 
mentioned before, we can use both quantitative and qualitative data. 

Quantitative : We can summarize by using statistical descriptions of activities and responses 
utilizing means, medians, and modes. When appropriate, you can use range or standard deviation. 

Qualitative : A written summary of observations can be presented, especially when there are not too 
many workers involved in a particular activity. Anecdotal information is welcome in this kind of 
document, which can give more of a “flavor” of the program than only a statistical description. 
Case-study summaries can also indicate other variables the program managers might want to look 
for in a further iteration of the program. 

In our working definition of “Evaluation,” we said that it was a process that judged programs 
against criteria, using evidence. However, judgment implies values. Whose values are we using? 

Within the workplace education mainstream, we are used to considering the values of the business 
community as our first priority. That is as it should be. However, it is also clear to us that, if we 
do not take into account the values of the workers (recipients of the educational activity), the 
program will stumble and fall. Other stakeholders, too, may have values that need to be 
recognized. Productivity may not turn out to be the bottom line value of your community college, 
but quality education might. Value judgments are basic to the needs assessment process, and we 
fervently hope that they are the basis for the written objectives of the program. Throughout the 
program, from needs assessment to evaluation, stakeholder values are the basis for action. Your 
role, as an evaluator, is to be sensitive to stakeholder values as they are expressed in assumptions 
about the outcomes of workplace education. 

For example, in a community college that is just beginning to develop a workplace education 
program in its community, an important value may be the good will of local businesses. The local 
workforce, faced with a volatile marketplace, desires and values education which will help them be 
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competitive for new kinds of jobs. Businesses may have their primary focus on increasing their 
day-to-day productivity and desire very specific technical training. All stakeholder concerns need 
to be recognized and addressed as much as possible in the final product. 

■s’Now that we have described the outcomes of our workplace programs and listed the 
inputs, we need to look at connections between them to discover any causal relations. 

We have already mentioned a few ways that we might look at associations between inputs and 
outcomes, but we must also be careful not to assume too much. For example, if we are evaluating 
an activity in which the instructor teaches a department to operate a new software system and, by 
George, six months later, they are doing fine; the question we may ask is how fine would they have 
done by simply working on their own and asking colleagues questions when they got stuck. 

Unless the program has been designed up-front with a control group, our outcomes may not really 
be as spectacular as we have calculated. 

This is one of those slippery slopes that we mentioned earlier. Even though we know that 
correlation doesn’t imply causality, if we have a high level of correlation... well, we need to take it 
seriously. If we find that the more hours workers spend in math class, the more math errors they 
make on the job, this correlation suggests that the needs assessment needs to be improved. Or, 
that the curriculum is not tailored specifically to the workplace. Or, that the instructor is not 
communicating to the workers. Something is broken and needs to be fixed. Although this may 
seem simplistically obvious to us, not all workplace evaluators may be conceptualizing in quite, that 
way. 

ktNow that we have described and looked at possible connections, we need to look at how 
the objectives of the workplace program were carried out in relation to what went into the 
program as well as the outcomes. 

This part of the evaluation has to do mainly with the instruction. If the program had a firm grip on 
the objectives, did the curriculum, course outlines, and lesson plans reflect them? Were the right 
materials and resources in the right place at the right time? Were the instructors adequately trained 
both in the content and in educational methodology? 

These connections between the objectives and the educational delivery methodology are important. 
This is the place where you may find out that specific educational theory (particularly in a 
workplace environment) may not support the outcomes that the program wanted. For example, a 
blueprint reading curriculum may have been customized for a certain group of workers. The 
instructor may be uncomfortable with the curriculum and decides to teach blueprint reading in a 
way that he/she is used to and is comfortable with. To what extent are the outcomes reflective of 
a) the curriculum or b) the instructor? 

Specifically, how were they carried out in relation to the desired outcomes? Is there any 
interconnect with those relationships we found in the previous step? 
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Let’s say that a new employer has come to town and wants her employees trained before they 
begin to work. Her business will need fifty persons. In the initial meeting with workplace 
education staff, she says that she wants her employees to be trained in “effective verbal 
communication skills.” Now, you and I may have very different ideas of what “effective verbal 
communication skills” are. We hope that, in the original needs assessment, the concept has been 
operationalized into measurable behaviors, such as: listening skills, expressive skills, and conflict 
resolution skills (and we hope that each of these has been further delineated into specific behaviors 
with specific standards of performance). If you find, as an evaluator, that they have not been, you 
may have to query the employer and the workers about their expectations of the program. 

Nevertheless, back to the HomePage... 

This employer’s need for her employees to communicate more effectively must be translated into 
educational objectives for the training program. These objectives must be observable and the 
stakeholders must agree that they accurately reflect the training goals. Insofar as the training 
program has operationalized the needs statements into behaviorally-based objectives, you, as the 
evaluator, can judge whether the program, indeed, carried them out. For example, you may want 
to set up “matching” columns in which the expressed needs of the company, the learner, and other 
stakeholders were identified and the subsequent objectives of the program were translated into 
actual training. 

fcsThe next piece of the puzzle to look at is how the objectives are arranged within their 
hierarchies. Remember macro and micro objectives? Chinese boxes? 

You can implement different levels of evaluation at this point. We can label the first level 
“Reactions.” (Likert scales do well on this portion of the evaluation.) Some of us call them smile 
sheets, although we don’t necessarily expect smiles all the time! Some of the “reactions” you may 
want are: 

♦ how did the workers react to the instructor, the material presented, the methods 
used in training, the place where the training took place, the other workers involved 
in the training? 

♦ how did the supervisors react to the program? 

The next level of evaluation we can call “Learning.” This is where you can use any quantitative 
techniques in order to measure actual changes of behavior from pre-training to post-training state. 
We are looking at primarily cognitive changes at this point. For example, the electricians are able 
to understand and to solve problems using Ohm’s Law. One of the ways that “learning” can be 
measured is: 

♦ paper-and-pencil tests at the beginning, during, and after training (to measure actual 
change) 

♦ hands-on performance tests on the work site, using a check list (if you’re going to 
evaluate this way, you may want to make sure a “before” version has been used) 
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The third level of evaluation has to do with “job performance.” This is where you have to find out 
if the training has actually impacted on the day-to-day job performance of the workers that already 
have been trained. Some of the ways you can do that are: 

♦ participant observation teams, in which workers can be observed using the new 
behaviors 

♦ interviews with supervisors 

♦ check-off lists for supervisors 

The fourth level of evaluation can be called “organization,” or the impact of the program on the 
company as a whole. This level can utilize all the foregoing data and then go on to analyze how 
the improved performance of the workers has impacted the company’s functioning in a positive 
way. Impact evaluation has always been the most troublesome for workplace educators. Like 
programs, this level of evaluation must be “customized” for the evaluation. If the company has a 
product, can the training be demonstrated to have affected its production? If the company is 
service-oriented, can the training be demonstrated to have affected the quality of service delivery? 

At this point, you can look at the program and show how these four levels of evaluation work 
together. The first two can help decision-makers change or modify the program, or simply 
congratulate themselves on the way the program is coming along; the last two can help the 
company decide on whether or not the program is doing what it was intended to do (impacting the 
organization in a positive way). 

■sThe last (yes, finally) step would be to include information on how well the needs 
assessment, the delivery of instruction, and consequent outcomes worked together in an 
integral whole. 

Was the needs assessment translated into a program that produced outcomes that satisfied 
the needs assessment ad infinitum. 

Congratulations! ! ! 
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Chapter Four 

We have now come full circle — from the first sentence in the Preface (evaluation is a statement of 
accountability) to the end of this manual — where we are going to talk a bit about the whole issue 
of accountability and how to handle it within a workplace education evaluation framework. 

What is accountability? 

For our purposes, let me put out the following operational definition. 

“Accountability refers to the process of reporting efficiency of planned program operation, 
primarily to the learners and leaders of the target public, the organization, funding sources, 
the profession, and, where appropriate, the governance body.” 

Boone, E.J.(1985) Developing Programs in Adult Education , p. 196 

If we take this approach, then business and industry is not our only “customer.” As professionals, 
we have responsibilities to a variety of stakeholders. Our business and industry partners know full 
well the importance of accountability. If their products are not up to standard or if their services 
are not delivered promptly and well, their profit margin will decline or their funding will diminish. 
As workplace education professionals, we have no less a responsibility to report accurately on the 
results of our efforts. 

What are we accountable for? 

Everything, from planning to inputs to outcomes. Did what we put into the program result in 
accomplishing what we wanted to see come of it? 

What can we do to make sure that we are accountable workforce education professionals? 

Three tasks should do it (you can add more if you like). 

...report results to all the stakeholders 

...analyze how the workforce program has added positively to the effort for change within 
the business or industry. (If the company wasn’t interested in change, we wouldn’t be 
there.) 

...recommend how the company can further its efforts toward change 

Reporting results 

Report first to the workers. This is what we said we would do and this is what we did. 

Give the company enough data so that they can make a rational decision on what to do next. 

Tell the company if any of its management practices affected the training. 

Attach results to dollars expended (especially for funding sources). 
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How Do We Document Our Results? 

Statistical summaries of participation 
Summaries of outcomes 
Narratives about unexpected observations 
Tell about marketing techniques, especially if they were unusual 
Discuss instructor’s creative techniques: 
methods of teaching 
materials used 
classroom anecdotes 

If you have a matrix of needs and objectives, add that to your documentation. 

Go one step further and discuss how these needs and objectives may be used in other companies 

Include any diagnostic and evaluation instruments 

Add any planning instruments or other useful program documents 

Write a section as a case study, telling how the program evolved 

If anything happened in your program that might be useful to other workplace education programs, 

tell us! 

Speaking of that, don’t forget to share your experiences with the rest of us at conferences and in 
journals. 

Analyzing How the Program Has Affected the Company 

We mentioned before that if the company didn’t want change, we wouldn’t be there. So, in this 
part of the evaluation, it is necessary for you to determine how the program has impacted that need 
for change in the company. It may be helpful to think about the following questions. 

What is the company’s purpose? 

What is its general philosophical orientation? (Here is where the company’s value system and 
ethics statement would be useful to review.) 

What does it do, how is it structured, and what kinds of processes are in place? 

The company does not exist alone on an alien planet. It exists within a community that has a 
particular economic and social structure. The company exists in a community in a certain historical 
time period. The historical community has its own value system, structures, and processes. How 
do these two systems (community and company) act and react together? How did these structures 
and interactions affect the results of your workplace education program? What else needs to be 
taken into account in the next phase of training? 

The worker, in some sense, is shaped by the company. The company has an overall interest in 
producing a product. That product changes the community. The community, in turn, produces 
the workers for the company. 

To what extent, then, has your program changed the behavior of the workers whose behavior then 
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moved the company closer to its goals? 

How Can the Company Better Meet the Needs of the Community 

The company knows that if it does not meet its community’s needs, its own goals (bottom line: 
survival!) will not be met. Your recommendations are the last piece of the evaluation puzzle. 

How effective is the company in achieving its goals? 

How is it impacting on the community? 

What kinds of workplace education programs will help the company achieve its goals more 
successfully? 

What is the company’s ideal vision of what it wants to be? 

How can we, as workplace education professionals, assist it to achieve that vision? 

For example, let’s take Company A. After your evaluation, you find that very few of the 
objectives of the program have been met. What are your recommendations? You might want to 
look at the link between needs and objectives. You would suspect that the needs assessment was 
not adequate or that the objectives did not flow naturally from the assessment. Back to planning. 

Company B did better, but you think that the program has room to improve. What are your 
recommendations? Look at the programmatic procedures: who made decisions? who reported to 
whom? what tasks were delegated? how centralized was decision-making? Back to 
implementation. 

Company C turned out great! Objectives have been achieved! All is well! What are your 
recommendations? Now that you have changed the character of the workers (and thereby the 
company), it’s back to the drawing board and begin the process over again. Back to needs 

assessment. 

Conclusion 



The thinking on workplace evaluation systems continues to evolve. This is one approach that has 
had some success in keeping us focused on what we were trying to achieve. You may discover 
that other systems work better for you in your particular circumstance. You may end up by 
designing systems of your own. In any case, let’s keep the conversation going! 
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